{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R6gHiH-I7uFa"
   },
   "source": [
    "# Improving Computer Vision Accuracy using Convolutions\n",
    "\n",
    "In the previous lessons you saw how to do fashion recognition using a Deep Neural Network (DNN) containing three layers -- the input layer (in the shape of the data), the output layer (in the shape of the desired output) and a hidden layer. You experimented with the impact of different sized of hidden layer, number of training epochs etc on the final accuracy.\n",
    "\n",
    "For convenience, here's the entire code again. Run it and take a note of the test accuracy that is printed out at the end. \n",
    "\n",
    "# 利用卷积提高计算机视觉精度\n",
    "\n",
    "在前几节课中，你看到了如何使用包含三层的深度神经网络（DNN）进行时Fashion MNIST图像识别--输入层（根据输入数据的形状）、输出层（根据类别数量）和一个隐藏层。你实验了不同大小的隐藏层、训练次数等对最终精度的影响。\n",
    "\n",
    "为了方便起见，这里又提供了完整的代码。运行它，记下最后打印出来的测试精度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "height": 207
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 19572,
     "status": "ok",
     "timestamp": 1550247198665,
     "user": {
      "displayName": "Laurence Moroney",
      "photoUrl": "https://lh3.googleusercontent.com/-RcxktLY-TBk/AAAAAAAAAAI/AAAAAAAAABY/b4V4dTIqmPI/s64/photo.jpg",
      "userId": "06401446828348966425"
     },
     "user_tz": 480
    },
    "id": "xcsRtq9OLorS",
    "outputId": "027ddd16-b2d9-41a0-85aa-9da6275085e9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1875/1875 [==============================] - 2s 1ms/step - loss: 0.4969 - accuracy: 0.8265\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 2s 1ms/step - loss: 0.3727 - accuracy: 0.8669\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 2s 1ms/step - loss: 0.3372 - accuracy: 0.8781\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 2s 1ms/step - loss: 0.3131 - accuracy: 0.8859\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 2s 1ms/step - loss: 0.2955 - accuracy: 0.8921\n",
      "313/313 [==============================] - 0s 897us/step - loss: 0.3591 - accuracy: 0.8744\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "mnist = tf.keras.datasets.fashion_mnist\n",
    "(training_images, training_labels), (test_images, test_labels) = mnist.load_data()\n",
    "training_images=training_images / 255.0\n",
    "test_images=test_images / 255.0\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Flatten(),\n",
    "  tf.keras.layers.Dense(128, activation=tf.nn.relu),\n",
    "  tf.keras.layers.Dense(10, activation=tf.nn.softmax)\n",
    "])\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(training_images, training_labels, epochs=5)\n",
    "\n",
    "test_loss = model.evaluate(test_images, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zldEXSsF8Noz"
   },
   "source": [
    "Your accuracy is probably about 89% on training and 87% on validation...not bad...But how do you make that even better? One way is to use something called Convolutions. I'm not going to details on Convolutions here, but the ultimate concept is that they narrow down the content of the image to focus on specific, distinct, details. \n",
    "\n",
    "If you've ever done image processing using a filter (like this: https://en.wikipedia.org/wiki/Kernel_(image_processing)) then convolutions will look very familiar.\n",
    "\n",
    "In short, you take an array (usually 3x3 or 5x5) and pass it over the image. By changing the underlying pixels based on the formula within that matrix, you can do things like edge detection. So, for example, if you look at the above link, you'll see a 3x3 that is defined for edge detection where the middle cell is 8, and all of its neighbors are -1. In this case, for each pixel, you would multiply its value by 8, then subtract the value of each neighbor. Do this for every pixel, and you'll end up with a new image that has the edges enhanced.\n",
    "\n",
    "This is perfect for computer vision, because often it's features that can get highlighted like this that distinguish one item for another, and the amount of information needed is then much less...because you'll just train on the highlighted features.\n",
    "\n",
    "That's the concept of Convolutional Neural Networks. Add some layers to do convolution before you have the dense layers, and then the information going to the dense layers is more focussed, and possibly more accurate.\n",
    "\n",
    "Run the below code -- this is the same neural network as earlier, but this time with Convolutional layers added first. It will take longer, but look at the impact on the accuracy:\n",
    "\n",
    "你的训练准确率大概是89%，测试准确率大概是87%......还不错......但是如何让它变得更好呢？一种方法是使用一种叫做Convolutions的方法。这里不打算纠结于卷积的细节，而想抓住它的核心思路，即通过卷积操作缩小了图像的内容，将模型注意力集中在图像特定的、明显的特征上。\n",
    "\n",
    "如果你曾经使用滤镜进行过图像处理（比如：https://en.wikipedia.org/wiki/Kernel_(image_processing)），那么convolutions看起来会非常熟悉。\n",
    "\n",
    "简而言之，如果取一个二维数组（通常是3x3或5x5）并将其应用到图像上。通过根据该矩阵内的公式改变底层像素，就可以进行图像边缘检测等工作。例如上面的链接，会看到一个3x3的矩阵，它是为边缘检测而定义的，其中间的单元格是8，而所有相邻单元格都是-1。在这种情况下，对于每个像素，把它的值乘以8，然后减去它周边像素的值（因为每个都乘了-1）。扫描整个图像，对每个像素都这样做，最终会得到一张边缘被增强的新图像。\n",
    "\n",
    "这种计算对于计算机视觉来说是非常理想的，因为通常情况下，能够像这样被突出显示的特征才是区分一个物品和另一个物品的关键。卷积使得所需要的信息量会少很多......因为只需要对突出显示的特征进行训练。\n",
    "\n",
    "这就是卷积神经网络的概念。在全连接层之前，增加一些层来做卷积，那么输入全连接层的信息就会更加集中，也可能更加准确。\n",
    "\n",
    "运行下面的代码--这和前面的神经网络是一样的，但这次先加了卷积层。这会花费较长的时间，但看看对精度的影响。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "height": 605
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 29440,
     "status": "ok",
     "timestamp": 1550247270616,
     "user": {
      "displayName": "Laurence Moroney",
      "photoUrl": "https://lh3.googleusercontent.com/-RcxktLY-TBk/AAAAAAAAAAI/AAAAAAAAABY/b4V4dTIqmPI/s64/photo.jpg",
      "userId": "06401446828348966425"
     },
     "user_tz": 480
    },
    "id": "C0tFgT1MMKi6",
    "outputId": "b9c48f3c-639a-4c14-ebbe-657cacca81f8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2.0\n",
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 64)        640       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 13, 13, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 11, 11, 64)        36928     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 5, 5, 64)          0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 1600)              0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 128)               204928    \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 243,786\n",
      "Trainable params: 243,786\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/5\n",
      "1875/1875 [==============================] - 35s 19ms/step - loss: 0.4311 - accuracy: 0.8443\n",
      "Epoch 2/5\n",
      "1875/1875 [==============================] - 35s 19ms/step - loss: 0.2897 - accuracy: 0.8949\n",
      "Epoch 3/5\n",
      "1875/1875 [==============================] - 35s 19ms/step - loss: 0.2473 - accuracy: 0.9079\n",
      "Epoch 4/5\n",
      "1875/1875 [==============================] - 35s 19ms/step - loss: 0.2162 - accuracy: 0.9205\n",
      "Epoch 5/5\n",
      "1875/1875 [==============================] - 35s 19ms/step - loss: 0.1914 - accuracy: 0.9292\n",
      "313/313 [==============================] - 2s 5ms/step - loss: 0.2541 - accuracy: 0.9108\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)\n",
    "mnist = tf.keras.datasets.fashion_mnist\n",
    "(training_images, training_labels), (test_images, test_labels) = mnist.load_data()\n",
    "training_images=training_images.reshape(60000, 28, 28, 1)\n",
    "training_images=training_images / 255.0\n",
    "test_images = test_images.reshape(10000, 28, 28, 1)\n",
    "test_images=test_images/255.0\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "  tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "  tf.keras.layers.MaxPooling2D(2,2),\n",
    "  tf.keras.layers.Flatten(),\n",
    "  tf.keras.layers.Dense(128, activation='relu'),\n",
    "  tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model.summary()\n",
    "model.fit(training_images, training_labels, epochs=5)\n",
    "test_loss = model.evaluate(test_images, test_labels)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uRLfZ0jt-fQI"
   },
   "source": [
    "It's likely gone up to about 93% on the training data and 91% on the validation data. \n",
    "\n",
    "That's significant, and a step in the right direction!\n",
    "\n",
    "Try running it for more epochs -- say about 20, and explore the results! But while the results might seem really good, the validation results may actually go down, due to something called 'overfitting' which will be discussed later. \n",
    "\n",
    "(In a nutshell, 'overfitting' occurs when the network learns the data from the training set really well, but it's too specialised to only that data, and as a result is less effective at seeing *other* data. For example, if all your life you only saw red shoes, then when you see a red shoe you would be very good at identifying it, but blue suade shoes might confuse you...and you know you should never mess with my blue suede shoes.)\n",
    "\n",
    "Then, look at the code again, and see, step by step how the Convolutions were built:\n",
    "\n",
    "模型在训练数据上的精度可能上升到93%左右，在验证数据上可能上升到91%。\n",
    "\n",
    "这是朝着正确方向取得的显著进步!\n",
    "\n",
    "试着运行更多的epochs--比如20个epochs，然后观察结果! 虽然结果可能看起来非常好，但实际上验证结果可能会下降，这是因为\"过拟合\"造成的，后面将会讨论。\n",
    "\n",
    "(简而言之，'过拟合'发生在网络模型从训练集中学习到的结果非常好，但它太狭隘了，只能识别训练数据，而在看到*其他*数据时效果不佳。举个例子，如果你一辈子只看到红色的鞋子，那么当你看到一双蓝色的麂皮鞋可能会感到迷惑......再举一例，应试教育往往使得学生只对做过的题目有很好的正确率，但对真实的问题却错误率很高）\n",
    "\n",
    "接下来，观察程序代码，看看卷积模型是如何一步步建立的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RaLX5cgI_JDb"
   },
   "source": [
    "Step 1 is to gather the data. You'll notice that there's a bit of a change here in that the training data needed to be reshaped. That's because the first convolution expects a single tensor containing everything, so instead of 60,000 28x28x1 items in a list, we have a single 4D list that is 60,000x28x28x1, and the same for the test images. If you don't do this, you'll get an error when training as the Convolutions do not recognize the shape. \n",
    "\n",
    "第一步是收集数据。你会注意到，这里和之前有一点变化，训练数据需要改变维度（shape）。这是因为第一次卷积期望一个包含所有数据的单一张量，所以要把训练数据设置为60000x28x28x1的一个4D列表，测试图像也是如此处理。如果不这样做，会在训练时得到一个错误，因为卷积操作将不能识别数据形状。\n",
    "\n",
    "```\n",
    "import tensorflow as tf\n",
    "mnist = tf.keras.datasets.fashion_mnist\n",
    "(training_images, training_labels), (test_images, test_labels) = mnist.load_data()\n",
    "training_images=training_images.reshape(60000, 28, 28, 1)\n",
    "training_images=training_images / 255.0\n",
    "test_images = test_images.reshape(10000, 28, 28, 1)\n",
    "test_images=test_images/255.0\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SS_W_INc_kJQ"
   },
   "source": [
    "Next is to define your model. Now instead of the input layer at the top, you're going to add a Convolution. The parameters are:\n",
    "\n",
    "1. The number of convolutions you want to generate. Purely arbitrary, but good to start with something in the order of 32\n",
    "2. The size of the Convolution, in this case a 3x3 grid\n",
    "3. The activation function to use -- in this case we'll use relu, which you might recall is the equivalent of returning x when x>0, else returning 0\n",
    "4. In the first layer, the shape of the input data.\n",
    "\n",
    "You'll follow the Convolution with a MaxPooling layer which is then designed to compress the image, while maintaining the content of the features that were highlighted by the convlution. By specifying (2,2) for the MaxPooling, the effect is to quarter the size of the image. Without going into too much detail here, the idea is that it creates a 2x2 array of pixels, and picks the biggest one, thus turning 4 pixels into 1. It repeats this across the image, and in so doing halves the number of horizontal, and halves the number of vertical pixels, effectively reducing the image by 25%.\n",
    "\n",
    "You can call model.summary() to see the size and shape of the network, and you'll notice that after every MaxPooling layer, the image size is reduced in this way. \n",
    "\n",
    "接下来是定义模型。首先要添加一个卷积层。参数是\n",
    "\n",
    "1. 你想要生成的卷积数（过滤器数量）。这个数值是任意的，但最好是从32开始的倍数。\n",
    "2. 卷积的大小（过滤器的大小），在本例中为3x3网格。这是最常用的尺寸。\n",
    "3. 要使用的激活函数 -- 在本例中，我们将使用relu，你可能还记得它相当于当x>0时返回x，否则返回0。\n",
    "4. 在第一层，设定输入数据的形状。\n",
    "\n",
    "在卷积层之后加上一个MaxPooling层，用来压缩图像，同时保持卷积所强调的特征内容。通过为MaxPooling指定(2,2)，效果是将图像的大小缩小四分之一。它的想法是创建一个2x2的像素数组，然后选取最大的一个，从而将4个像素变成1个，在整个图像中重复这样做，这样做的结果是将水平像素的数量减半，垂直像素的数量减半，有效地将图像缩小25%。\n",
    "\n",
    "可以调用model.summary()来查看网络的大小和形状，你会注意到，每一个MaxPooling（池化）层之后，图像的大小都会以这种方式减少为原来的1/4。\n",
    "\n",
    "```\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RMorM6daADjA"
   },
   "source": [
    "Add another convolution\n",
    "\n",
    "再增加一个卷积层和MaxPooling2D\n",
    "\n",
    "\n",
    "```\n",
    "  tf.keras.layers.Conv2D(64, (3,3), activation='relu'),\n",
    "  tf.keras.layers.MaxPooling2D(2,2)\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b1-x-kZF4_tC"
   },
   "source": [
    "Now flatten the output. After this you'll just have the same DNN structure as the non convolutional version\n",
    "\n",
    "现在对输出进行扁平化处理。在这之后，你将拥有与非卷积版本相同的DNN结构，即全连接神经元网络。\n",
    "\n",
    "```\n",
    "  tf.keras.layers.Flatten(),\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qPtqR23uASjX"
   },
   "source": [
    "The same 128 dense layers, and 10 output layers as in the pre-convolution example:\n",
    "\n",
    "含有128个神经元的全连接层，以及10个神经元的输出层。\n",
    "\n",
    "```\n",
    "  tf.keras.layers.Dense(128, activation='relu'),\n",
    "  tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C0GSsjUhAaSj"
   },
   "source": [
    "Now compile the model, call the fit method to do the training, and evaluate the loss and accuracy from the test set.\n",
    "\n",
    "现在编译模型，调用model.fit方法做训练，接着用测试集评估损失和准确率。\n",
    "\n",
    "```\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(training_images, training_labels, epochs=5)\n",
    "test_loss, test_acc = model.evaluate(test_images, test_labels)\n",
    "print(test_acc)\n",
    "```\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IXx_LX3SAlFs"
   },
   "source": [
    "# Visualizing the Convolutions and Pooling\n",
    "\n",
    "This code will show us the convolutions graphically. The print (test_labels[;100]) shows us the first 100 labels in the test set, and you can see that the ones at index 0, index 23 and index 28 are all the same value (9). They're all shoes. Let's take a look at the result of running the convolution on each, and you'll begin to see common features between them emerge. Now, when the DNN is training on that data, it's working with a lot less, and it's perhaps finding a commonality between shoes based on this convolution/pooling combination.\n",
    "\n",
    "# 将卷积和池化的结果可视化\n",
    "\n",
    "这段代码将以图形方式向我们展示卷积的结果。print(test_labels[;100])向我们展示了测试集中的前100个标签，你可以看到索引0、索引23和索引28的标签都是相同的值（9），它们都是鞋子。让我们来看看对图像做卷积操作的结果，会看到它们之间的共同特征出现。之后，当DNN在该数据上进行训练时，模型训练的工作内容就少了很多，模型会在卷积/池化的基础上找到鞋子图像的共性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 400,
     "status": "ok",
     "timestamp": 1549465887353,
     "user": {
      "displayName": "Laurence Moroney",
      "photoUrl": "https://lh4.googleusercontent.com/-wUzpekukCVw/AAAAAAAAAAI/AAAAAAAAAHw/pQPstOOJqqE/s64/photo.jpg",
      "userId": "17858265307580721507"
     },
     "user_tz": 480
    },
    "id": "f-6nX4QsOku6",
    "outputId": "6b85ed93-6868-4c2c-b066-0808d6536878"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9 2 1 1 6 1 4 6 5 7 4 5 7 3 4 1 2 4 8 0 2 5 7 9 1 4 6 0 9 3 8 8 3 3 8 0 7\n",
      " 5 7 9 6 1 3 7 6 7 2 1 2 2 4 4 5 8 2 2 8 4 8 0 7 7 8 5 1 1 2 3 9 8 7 0 2 6\n",
      " 2 3 1 2 8 4 1 8 5 9 5 0 3 2 0 6 5 3 6 7 1 8 0 1 4 2]\n"
     ]
    }
   ],
   "source": [
    "print(test_labels[:100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 349
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1429,
     "status": "ok",
     "timestamp": 1549466091111,
     "user": {
      "displayName": "Laurence Moroney",
      "photoUrl": "https://lh4.googleusercontent.com/-wUzpekukCVw/AAAAAAAAAAI/AAAAAAAAAHw/pQPstOOJqqE/s64/photo.jpg",
      "userId": "17858265307580721507"
     },
     "user_tz": 480
    },
    "id": "9FGsHhv6JvDx",
    "outputId": "e144d639-cebc-4d0a-9c7a-8571f70d6159"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWcAAAD7CAYAAAC2a1UBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO29e5hlVXXo+xtr7feud3dXv6FBEUR8gIgg6mlDHigeNefkI+A1l3M1nycn5l794vkUc25iruckIcm9fnqMJqJywPiC+IIkBEUU8Q1NAwo0NK9u6Ka6q7u667Wr9mOtNe4fe1exu/aqqv1+VI3f99W39x5rrjXHmrX3mHONOecYoqoYhmEY3YXTaQUMwzCMSsw4G4ZhdCFmnA3DMLoQM86GYRhdiBlnwzCMLsSMs2EYRhfSkHEWkctF5HEReVJErm2WUoZhGOuduo2ziLjAp4E3A+cCV4vIuc1SzLDOzzDWM5EGzr0IeFJVnwYQka8BbwceXbYySWhc0g1U2dvM6YnjqrqpmrJlnd9vAIeA+0TkNlUNbV9r2+rbFoodH/BJwAU+r6rXrVJ+Xe/WUlVp1bXXe9sCod/dRozzduC5ss+HgNeudEJc0pyTuKKBKnubvfP/eLCG4jV1fta21bdtrR3fC7iNqNjD+G2oY722LYAf+t1t+YSgiLxXRPaIyB6PXKurW0uEdX7bO6TLWmOx41PVPLDQ8RlG19CIcT4M7Cz7vKMkOwVVvV5VL1TVCyPEG6jOWIp1fHVTVcdX3r5t02wNYHMlzaER43wfcJaInCEiMeAq4LbmqGVQRednHV9rKW/fTuvSK9hCgeZRt3FWVQ/4I+A7wD7gFlV9pFmKGdb5tZCqnvqMujCXUZNoZEIQVb0duL1JuhhlqKonIgudnwvcYJ1f01js+Cga5auAd3ZWpTVDzQsFjHAaMs5Ga7HOrzVYx9d5ROS9wHs7rUc3Y8bZWJdYx9cyql4oAFwPts55OSy2hmEYzcTmSpqEjZyNluNUOQZwQ8qdEe+rkM37lQOtvbWrZbQAcxk1DzPOTWbBEAUEFTLDWA+Yy6g5mHEOYSUDuyBzcBCK4QYUJSAgRYI+J0qgSkYL5PGIEyUuLg4tC01gGMYaxIzzCqw24l0wuD6Kg0NcXAYjLgVV8gWfoCTrc62ZDcOojXVpNaJEiOASxSHlRBARCkFAQQMi4hCVotGdDgpkyQOnjqKXEhDgiU+/28/pfTDvOxyfgmmZJcUISVdwbOBsGEYNrAvjXO6ScHCIEyXlROiPuGxPCTFHmcy7ZDyIu9AXAQUOzArjvodfMr7l13K0OHIOUApSwBOPTQmH12w4yUQuwQNTcFwOMxoMsyG+NoxzNb7zW//DQxWyT979hgqZF9LX/WwyUyH700t+VSH73IOvqDx5ZlXVDKOnWPPGOcygxMUl7br0R4XhmE/cUbzAJR8IEYGIKAFC1BHifpQcBYCSZ/nUazoIUY3iEiHpQl8sx5wfwUfIBbPMkWfej5vH2TCMmljTxnnpJF4gChowGo/yov6ATfE8L9twjHi0wN4j28hOxwkUZj1BgaGYQ38kwYl8lAPBPJ54RDRSuppLgBIlws5IiqQr7EwVGE5lyBRizIrHdPYgv4rO8sz8KGIrNgzDqIE1a5zDRsxFt4ZLfxS2JXNsTs2xc/MRovE8B0+OEHPi5API+qAK6SgkokoucCAPPh4ODi4OAQEuDhGE4ZjDQBSG43mSiSyxWZ+cZAmCGeZyM8zlDrS/AQzD6GnWrHFeYGGyLqIuA5omLi4b4gGjyTnyvssP97+UnO/y4MkEj89m8QnIUSBOlNfGk7xkIMvZg8pvShpIkw8cfBWSrk86WiDh5tgycIh0cp6J6UEeeX4Hz8ymmeLZTt+6YRg9zJo3zp74FCRPTNMMuzH6ow7bkzl2Dp3gsWNb+OwBn4POo0wXnidXOF46KyAeHeWy6L/n1VsPsfOMZ9lwxWFIpuDoBDoDMizo9u1oqo/89nMJ0i9h43+9k5v++TIOzuWZ8g519L5bgYR4zm960zOnfD42vrGizN8c/kyFbO6Rf1ch00isQhY/6/crZJ9wrllRT8NYC6x54+zg4KhDFJeRmMtADKJOwHwhxkwhyoTzPFP5QxS8KVSzi+d5foaTeYfxmQE2TPexISgtL/AV9R2kECDZecQrEM8/gDoOh46McjTrc4xp/MAykxiGUT9r0jifsppChajEGI2keMPmDFtTsxyd6+Onz2/niZkIx/2nyXvHUS2ccg3Pn+bmmfv44b4z2H34tfz+bJpUcp6p6ZeTySWIuj5R1yOTS3D34e0czLg8npnjl3oXeT+D551s920bhrGGWNU4i8gNwFuBcVU9ryQbAW4GdgEHgCtVtSus0dKJwOLIGZKucPrAFFuHJzh6sI9n5yKMzXtkvSmKCRuW4jOeuZdx7iWi7+TiwzsZjGc5kU2R8V5otqPZGJ+beIxDs3e39sYMw1hXVDNyvhH4O+CLZbJrgbtU9bpSAsdrgQ83X73qWWqUFzaNjNLPpniUbSlFVZjPJjiQSXLf9DTH3HE8f/XdC886j3PbodeSigySKSi54IWoaDO+xwn/QFPvZQEROUBxe4UPeJbLzjDWD6saZ1W9R0R2LRG/Hdhden8TcDctNs7lxrc8+NACS4MQ+XgEBGyKR3nVcIGBWNFtMZNN8viUsCf7TyVXhr9q3ccze7lZyqMenrq9balLpMm8SVWPr16sOqqNkBc2+bd7pL9CduYbTg3WmfjIXFXXT73sh1WVgzurLGcYa4t6fc6bVXWs9P4IsHm5guXpaGKSrrO6FyiPcbGwHXsBpTiidXBIaIIIDhvisD01hwgczfSR9V2OFwqo5oBqEzDoKZOFhmEYrabhCUFV1ZXSzJSno0k7G6qyhuU7+wJRHD11FBeUqnNUTjHQC2uakxpnV6yPwZjwxs0neP0FD/DcM6fxsT1n8iD7Oebtp3rD3FEU+G6pfT9bastFmt3xGYbRPdRrnI+KyFZVHRORrcB4M5VaysK268X3ZfIFw70walYCIrgMxoRNiYAt/dP07zxK6sgm9usYz2W+30pVm83rVfWwiIwCd4rIY6p6z8LBejo+wzB6g3qN823ANcB1pddbm6HMgrENC8+5YJSXHgsEtBSYaFgHGHEH2ZRw2L15ii19M8zmEvzgX3+dfSeHmeD+ZqjZNlT1cOl1XES+BVwE3LPyWUY12GRrOGPXvLRC9lv/8mTV54vIToqLBzZTfPK7XlU/2Sz91hPVLKX7KsXJv40icgj4KEWjfIuIvAc4CFzZqCKLxrdsZOyoFI216KJRLr5b6srwUAL6ZIQX9Qs7U3kuOWcfI6eNcdf338hNTw3zvDfLZP5Ao2q2DRFJA46qzpTe/ybwsRXOqJjsC+vkwib6clK5lPCdm4cqZB/4LzdUyN780XctkXx+eRW7j6ZOthoAeMAHVXWviPQD94vInar6aKcV6zWqWa1x9TKHLmumIgtGeEVdyoxN0RQ5RFXo0zQRXLb3RTirf57heI6ZmT704Daey/TxvDfLMXccVa+ZKreazcC3pBj4PwJ8RVXv6KxKhrEypYUCY6X3MyKyD9gOmHGukTbvEFx5dLd04m+xrAb48sKI2cEhplESxEhJhF3pKEMx5ZLRCS551YPMz6T5wSMv55nZJD8+XuChwp34uXn8YLbld9gsVPVp4JWd1mMNs+JkK5w64WrUTmkJ7vnAL0KOWduuQtdv3y5/DC8mVS2NmHFJSYS+iMumRMCmuMeW/in6th5DA+FINs7+aeE5Z4y8d5QeWZ1htI8VJ1vh1AnXlVYkGZWISB/wDeADqjq99Li17eq03TgLsrhRBAh1ZSwciyglQxyhT5O4CBtjMdIRYTSh7ErnGIjN89LRMYaHJzl0ZAtf/ebbOJKNc+fRAgfdpzjmPcl6MMwJx+UliYFTZG/aPF9R7s0XVE6KjrzkYIUsel6lv/qM/3RxhezQfE/5mBexydbWISJRiob5y6r6zU7r06t0bOS8dPVF+QSfT9E37EgxLkacKP1ulJTrcEYfbIoXOKN/hpfteJb+wRk2XfAYzjaHg5/axqeeneOo7ON4bh9+MNWZmzO6mtonW7uLgcTZofLp7ONVX2Nj+tWh8k3/6/+qkEVf86dVX1eKkyRfAPap6serPtGooK3GWYBIyS2R0FhJAUEQ4o6LKy8kc3JE6Is4JFxIRWAoGhBzA7YnswzFswzGs+QLUaYnB8j++FU4bsDe8S0cl33M+RMEaiE7jWWxydbWcSnwe8CvROTBkuxPVPX2DurUk7TZOAtxIqScCEnXJe7CQFSIOkXj2xf1iTpKyvWJOQE7B6YYSc+QTGTpH5jBjXhEEnnciM/JIxt59shWjsz2c/PBfn4VHGAi+DEz2YOo5tAqYmYY6xObbG0dqvpjsHzGzaC9xlmEmLgkS5mvEy4MRJW4owzHPQaiBaJOQDpSIBHxGB08yfDwJJF4nnjfHOIojls0uqrCZDbJeDbJQ/o0BzLfaeetGIaxxvmD0ffVfM7XMz+v+ZzjmXtD5W01zmlXuHAkwmiiwGhiHhGISMDCZK0jigjEHA9HlBMzA8zOp3h+aogHTw6S9YVCIPgKk3kYzxWY1HmO+I+sUvPaZ2vfHH/6uodOkWVziYpy+556UYXsLbeMVcgK3rGQWu6uVz3DMGqkrcY57gacPTjLzv4ptoxMoCoUvAi+75LJJcgWin5okWLs5elskrzv8rNjQ9wweQ+z+TH8YGaZ4PiGYRhrh7Ya54gE9EfzHJvr48hsPwV1mC1EKQQOc75DPjjVVZX1HQoBPD4TkPOnCTSPqvmSjfXNLudVofLXbvr1UPnnjn26QnY8Ex5nRv/1v1bKpmyHeydoq3GOuh6b+6f5zsHTuetYlmmZ5TAPkfNnKybwVF+IpeH583j+JMUg92t/zbJhGEZbjXOgDnP5OBM5h2fdg8z440xl95ubwjAMYwltNc5j81H+8pcbeNp9nKPZR/GD+VaneFo3PDI1y0v/9SedVsMwjCbRVuM8ExzjBz263dcwDKOdVJft0zAMw2gr1QTbD81sICIjwM3ALuAAcKWqnmydqoZhAPxy7uZQ+c8n/2eo/L+96/wK2a5bHggtu/8z51TIcuN7atDOaBbVjJwXMhucC1wMvE9EzgWuBe5S1bOAu0qfDcMwjCZQTSaU5TIbvJ1i+iqAmyhuH/twS7Rcw4jIDcBbgXFVPa8kq/mpJCUjnJO44hTZ3vmvtEDj2rkg+c4KWbfoZhjdSk0+5yWZDTaXDDfAEYpuD6N2bgQuXyKzpxLDWOdUvVpjaWaDUrhFAFRVl8tmUJ6ORoiQjJ+OH+QJtEAQ5PGDabphY4kjaVw3heDiOsVt5LnCBIFmWlqvqt5T6vTKsacSw+gwf3fkoprP+b/f9aOaz9mxzENkVcZ5mcwGR0Vkq6qOichWYDzs3PJ0NCl3RM9wX00mOktOZ5kLTjKdne+CTShCPLqBgeg2YpKiTwcJCDjI/WTzrTXOy1DVU0l5xxeTdJtUM7qVVKwyUH6t/OXPX1EhG8tYbtZOUM1qjeUyG9wGXANcV3q9ddVrqRAlRlwT+FIg6iRxnTR+0NkVfYJLzO0jJinimiSqMZSAmJMmJ5WR3epF6xiFr/RUUt7xpZ0NnX/8MAyjaVQzcg7NbEDRKN8iIu8BDgJXVlOho0KcBBGNkJJ+0snhUzJwd4qkpokGscXs3uAw6r6YwfT2ptXx3Oyd1Rat6qlkNcIm4rqFZuu2d/4fm3o9w+g01azWWCmzwWV1VaouEVziGidN9z6OD+ggMNi06z1XfdGan0oMo5sQERfYAxxW1bd2Wp9exHYIdhgR+SrwM+BsETlUehK5DvgNEXkC+PXSZ8PoJd4P7Ou0Er1Mx7JvG0VU9eplDtX1VGIYnUZEdgBXAH8B/HGH1elZzDgba5ZmbfBZT3z5ZGVgfmpPlvwJ4ENA/3IFylcaGeGYW8NYy9yIbfBpKyKy0BmGp1opoarXq+qFqnphm1TrOcw4G2sWVb0HOLFE/HaKG3sovb6jrUqtfS4F3iYiB4CvAb8mIl/qrEq9iRlnY71RddgBEXmviOwREQvLViWq+hFV3aGqu4CrgO+r6rs6rFZPYj5nY92y0gaf0vHFTT4rlTOMVmAjZ2O9cbS0sYdGNvgYq6Oqd9sa5/oR1fYNCETkGJABej3X+kbqu4fTVXVTs5WBxbY9WPpYr37dRK33ENq2paBS/1K2WuNvgQlVvU5ErgVGVPVDq128rH3XQttWy8K9tux7CxXf3bD6O0W76g//7rbTOAOIyJ5en6Ht9nvodv2qoRn3UNrgs5vij+wo8FHg28AtwGmUwg6o6tJJw5bq1St0+l7Xe/3mczbWLLbBx+hlzOdsGIbRhXTCOF/fgTqbTbffQ7frVw3deg/dqlcr6PS9ruv62+5zNgzDMFbH3BqGYRhdiBlnwzCMLqStxllELheRx0XkydIa065HRHaKyA9E5FEReURE3l+Sj4jInSLyROl1uAt07bn2hWL0OBEZF5GHy2TWvm2i0+2/WruKSFxEbi4d/0VIQuRG6g79fS8ps1tEpkTkwdLfnzWr/hVR1bb8AS7wFHAmEAMeAs5tV/0N6L0VuKD0vh/YD5wL/A1wbUl+LfDXHdazJ9u3pPsbgQuAh8tk1r7roP2raVfgD4F/KL2/Cri5ifWH/r6XlNlNcSNTW/8v7Rw5XwQ8qapPazHd9tcoRgjralR1TFX3lt7PUMzusJ3ui27Wk+0LPRM9rmfbdzU63P7VtGu5Ll8HLislnm6YFX7fHach41zjY952Tk2jd4guaYRqKT1OnQ/8ghqim7WJnm/fJVj7dpZ2tX817bpYRlU9YArY0GxFlvy+l3KJiDwkIv8mIi9rdt1h1G2cSwkcPw28meJj/tUicm6zFOs2RKQP+AbwAVWdLj+mxWefpq9JXKs+zlppVfsa1bEe2n+l3zewl2L8i1cCn6IYAqD1OpV8KrWfKHIJ8Oeq+lulzx8BUNW/Wq68S/yncak927aWPcC46uCI4ACuCI5A0g2IOIojAa4TkPciHM4XyAdL23h1Ik6aYSdNRBRHwBHIB0LGCwgIKvRZiaVBJuf0xHGtMoBMqfPbD/wGxdHEfcDVqvpoqN6S0GraNulU7tg/feuxCpn6boVsbqby+o/PTq1a55Zo5S0n3aBCNu1V1jnve6teH2prWyh2fMAnKfo8P6+qKybRjUtSk05l1qWpoLLtOsWIOxoqP+E3JfDeflU9uxkXKkdELhmORX66PRWv6byHJzPNVqWC5dpzJeps69DvbiOxNcIeR167tFB5rjBXIpyTuKKmShycRaPo4jIsSVKuS8IV+qNC0oWzB7KMxHOko3n6YjkOTA3z/zx3iIOz36v5poaSL+O3UxczElcGoj4JN+BQJsreyRwZ8ii6qE9Qsr4Ln8t1BnCWWPG98/8YFnlrORZ9cQAisuCLCzXOcUlX1bYvT1ZOul//gc9VyLzJvgrZgz+8uEJ26Y9uX7XO92z53QrZSwdnK2TfPzJQWWemuvR+tbRt2VPfYscnIrct1/EBJJ1+dqcr7+PWmb+vttqWc/lApX4AXzn5mQav7APc2uBFluO+7ak43979ippOevG3722ROi+wXHuuRH1t7Yd+d1se+EjLApannQ1VD9MdHPpJkHRcBqIumxJCf0R56eAsm9OziCgOStaLcGBmkCdn+pjIuRyZVybyHuPe/rr0nc4d5id6hKHZAc5Op9iaDIg48KqhBIUgwfGsMusH5IOAOS0QEJCjsGioF/Wvdni9PFV1fkZd1NTxGQCs+GRRL6rqvXy4ciBgNGacDwM7yz7vKMmagiAMRKIMRR22ppSXDc4wkpjj1a94mKGzDxLkogS5KJkjGznw49dzYNblgcxJHsh+E9Uc9brI8t4x9vl3EHH7iWauIOYk2BQPeMnAHAD7p1Mcz7nMFByCguJrgEcAWnOG4oYpfyqJ1eEuWsfU/NSXlPVtQLS2sKo1uYyMcBoxzvcBZ4nIGRSN8lXAOxtVKEmcETdOzHHYlBAGosppqRxnbRgnmcgye3KA/ANnMz+XYnK6nxOZPh6dSvDcXIFj7hFUCzQ2dxGgFPCDeY46J0hlRikEEYZiMRYyFSVcCFQINIqnCh7k8PAI8GtPI78cq3Z+9T6VGNVR3r5D7qi1bxXU4zIywqnbOKuqJyJ/BHyHYg95g6o+0qhCmyIJLtnk0x8pMBDLk4oUOGfbIV785p+hOZc7vvzb3HN0A8/MKnuDp5nXw8x491PwM/j+HDRsHBXVPL4WeHz+ezzlpDmNV6PsIh2BhKsMRZWhKARJyPvC8/NxpgsR5gKPWXIVPug6abjzOycxVCH7wvGLKmTBj75bIUv/uydCrri6fzmMv3iu0i/77vn3VcjO6Kv83509UDlivXms0l9dIy196lvnmMuoSTTkc1bV26n3F7uEhUm0VETYlswyGM8uHotEPCQF6ivj8yn2Twfs1zEOZu5CqW42v3aUIJghH8xwInaIyfxp+IGDxAQ3oqXVIkrEEaKOkHAdCuo2bcFRqzo/A6ij49vZP8f/u3tvhfzWVk2T1UHjE39NoWaX0bZkrD2a9RhdkQnFwaGPOCknwvnDHu946x1E+ub50pd/h396NsqmZzZx5gPnU1DhRxPzPOHsZaZwBG2eC2FFpnOH+Jn8kkgQJ5lL4xLB0VJnoinOTQ5wWtphbE6YzDdl4xLQ3M7PeAHr+DpPucvo5cN95jIKoSuMM0DKiTAUc3nJ4ATOu1+Jlxxm4oYY3537LMwB1a2oagmeP8FY5iehx+LRbZzLb7Ml4TFTiEC+zcoZdWEdX8swl1GT6ArjLAhDMZftKcgUohT+/lFQ4bHp7g9d4AfzPDOfxdMEkwUfXdsbqQxjNVqyUGA90hXGOYLDaWk4f2SKQ5k0777hbZwo5HmAH3RatVXx/El+kvs6biHJUPw0tsgZRLRyp1sn+PBrKudgtvXvqZCNZ8Im/5rHq1KVeVZvOP7pyoIhSejflPz9FmhktApzGTWPrjDOAAk3YCCWpTDTxyP+8xzlKWZznXoaWjCuAavP8Cl+MIUfTDEf6cdxXtRi3QyjuzGXUXPoCuPs4JB2A4ZSc8QmA04yxkzuOYKg9fvnK3Rx+hlKnIkjESazT+P51Tu7HYmS1BgOTnHXYHOW1BldQPzFo5zxrT+sPOCEBTALZ/z3zwqVj36+tU8uRm/SFcYZIOYGJONZoqLM+yfxg9UD6rSCiJNmk+wiohEy7rGajLMrUaIawQHyNG/VhmGsZeIvGmXXNyvXva/Esf9c9YbFRTZ97smayu/P1h44rZl0jXFWFVQdAiDQzo04/WCeCQ7hiEvBb//I3TAMA7rIOAeA7zv4jQcMagg/mGYi8xBA3euou8WZUShEK2TjmdZH81rKg3NfbXudhtHrdIVxDgjI+g6ZXIKs77Rtc0k4WveuQ18L5CjgILakzjCMhugK4+wRsH86Qj7Yzr5Jp2fdCZn8UZ5KPEKMFMM6SlxrCyBudC/P/2qWj+36eUPX+LefXbLMkeZMCF6YfFeofM/8l5pyfaO9dIVxVpTxrEchiHAoP0cQ9OY2Oz/IMJV7lpg7QF9kyIyzYRh10xXGOSDgWDDHyazDUWecQHOdVqk+1MMPsngStWV0hmE0RHcYZ1HG5GmmvSMUCplSTObeQ/HQYIYC4EmrouVVz8cfDFtXe2fb9WiEH8x/vkJ2QdJ2Axtrn44aZwcHF4dAFdeJ4kjdycANwzDWFB01zlFchp0Erggx/wwmnQ2ciB/hiH8C7UDaJ8MwjG5hVeMsIjcAbwXGVfW8kmwEuBnYBRwArlTVmoN6urgkXZeoA77GkGCQnGQRoqjF3jS6iKwv7J9ubA3+Bw+0Nv7Pmzf2h8r3PBcqNrqcavwINwKXL5FdC9ylqmcBd5U+10xcilm1t6fgFcMub9oY55Xu6bhuqp7LdRyRGBF3mHhkmIg2/lAiIgdE5Fci8qCIVIaTMwxjzbKqBVHVe0Rk1xLx24Hdpfc3AXcDH6618oTjsj3lMxIrsCM9y2j/NH2HdnLH4QEK3rFaL9dxXCdNOraVqJMkStNS77xJVUOCaa7OB8/fXyH7yvcb1qetjKReWSm0/T3GOqDe4d1mVR0rvT8CbF6uYHmusJikTzlWCAJmChEcIkScFHnf5Xgugt+jS+lUA7wghytRArGldIZRDU8/mOFdI9VH9wP4XmGwRdq8wFs2hbuJVmLPs82rv+Fnb1VVEVl2LFOeKyztbDil3JRmue9EDEeECbJMyyRTuo9cYbxRtTqCH8wyl8+Rd/sZim8D0queswoKfLfUvp8tteUiK3V8hmH0NvUa56MislVVx0RkK1CXNc1JgVkyFCTPc4WHmM832u2UT9h04tnXR9XHD5o2cn69qh4WkVHgThF5TFXvWTi4UsdnGEZvU69xvg24Briu9FpXgvgCeaadk+R1Di+Yr1MVl0RsK3F3gH53lFF/G7PODE9nf4rnT9R+NWeQDcmzERwmsk/UdY1moaqHS6/jIvIt4CLgnpXPMqpBRA4AM4APeKp64UrlT/rHuXnqhobqPJ65v6HzV+Ovnv9aS69fDSKyE/giRVenAter6ic7q1VvUs1Suq9SnPzbKCKHgI9SNMq3iMh7gIPAlfVU7onHjD9OPpjFD7L1XAKRKCPRM9io2zhTRjhng3AsO8zR4Cmm5ms3rInoRl4WvIKoONwbm2Wyjms0AxFJA46qzpTe/ybwsVqu8cDYjrrrD8v797aNQxWyjz3796te6x39/6VC9u2Z1c8DODH3UIXsvOSKdrQW6p5sNZbFAz6oqntFpB+4X0TuVNXKhJbGilSzWqPyV1rkskYrj2uCzc6L8B2Po7KfTC5AKaBayxrngJzOMivTnPD6GJtLMFkIKARzdenkBfOcdDNENUKh7tE8OOrglFYq1hlnYzPwLRGB4v/pK6p6R90KGUYbKC0UGCu9nxGRfcB2wIxzjXR0h+AWBnn54AYSrvLT41v5ZfJ+5v2TzOWegypjOqsWODG/j0l5mkNOkvuDNL7m6p5UzBfGeTj4LiIOnl9fqizBIUqMBDEKeHUZZ1kVrfsAAB7CSURBVFV9GghZR2Y0iRUnW+HUCVcs7VjNlJbgng9ULMUob9uU1L4qYj3QUeMcdxy2JT2Srs9INE7c78NzcghSw3SeoprF1yx+MEW+wXhDitcUP3NEIzgIjv2ou5UVJ1vh1AlXEdcmXGtARPqAbwAfUNWKZHzlbTsS2WxtG0JHjbMrQsL1Sbg+877PicIzeH6mw5lQmkMxqJPQyfh6I/H6/PgAz4Y8hf7o2Gvquta3ZyoGpQ0RbUKALJtsbR0iEqVomL+sqt/stD69Smej0gnEnICY6zMXeGTzhzqpTtNxRHDUIu11G/VNtgao1t/ZAexOvidUfvf8Fxq67gLffc0bQ+W/9vO6FlPVhRQnSb4A7FPVj7et4jVIR42zrzDjRfBVyNWZtw+KMS3AwZE4rpMk0AK+P1V3LsBGUfWYdk4igYMjDk6Hk9YaFdhka+u4FPg94Fci8mBJ9ieqensHdepJOmqc53yfQ5koESfKpDO2+gmhuMQiG4m7/QxEtrIx2MqczNa9zrnIgjGtzxUWaJbD8/dzxEkyEjuDzZxmBrqLsMnW1qGqP8ZmT5tCR42zpwEznuAK5Kj/kdGRCI5EiWmStCYICJCOBu738f0Z/GCefHTZsCOGYQAn/XG+NvmZms757Dn/R831/OfHatsEtHvb8zXX8bFuiq3RCFPMcf/sHJ74HPUrI6hVR0CuMEHBnyHrTjLhHsTzc3j+TAOaNT55rPigiq+Fjo0jXro9JJDvg5WiMMI2f/yASll1NHeC96T2ZnZ2w6iFjhrnWWeGw96jFPwMBX+yzqsogWYI/Ayef7KB8XezUcBHLdGrYRh10FHjPBgMsc3ZTeDAw7E9oaO13sQl4g7gSJyEM2Dxhw0Arjot/Itw9+PNuf4Xn6h/u77RfXTUOJ8Z6+d3Tp+kL5bjk/su5o66H5u7C0cSjCbOY5CNxc0oNhloGEaNdNQ4RxxhMJGlPzFPwllb64EdcYkEEaSqTGCGYRin0lHjPFnw+cX4KDEn4FmvvjgW3UigWcazj3HCPcBQZCej7OjI6Pnj974qRPqTuq8nkqiQVbMxQ0K+Zp1ag24YvUJHjXPGL7B/OoGDyzH3SCdVaTI+ee8IeQ+iTgrEfIGGYdRGZ1drkOPp+YCAgGmaZZwb20DSvGsYhmHUTzXB9kMzG4jICHAzsAs4AFypqidrqfyEM8Gh3AN4/gxBg3ELiriIuEAxlGh9xlUQStfAr/MahlHJU7OVbqFmcuPEp0Pl14y8r0J204nwskb3UM1s1UJmg3OBi4H3ici5wLXAXap6FnBX6XONlTu4TgzHiS0a1bWFrDohKCI3iMi4iDxcJhsRkTtF5InS63DLVTUMo6tY1Tir6piq7i29nwEWMhu8HbipVOwm4B21Vr45GOXNiSt4W/oqtqTqC0d5Kj6qhQZGzQCK4jc4ahYcSeM6A0Sc+GqFbwQuXyJruOMDeHF/oeKvEVSzFX9VnYdX8WcYxsrUtM5rSWaDzaWUNABHKLo9aqLfifHSQXjlsLJJmzVppjTuimj8GiIRXCeJS3TlmooB3k8sETfc8RmG0dtUPSG4NLNBKdwiAKqqpXQ/YectpqOJSfqUY74qGc8h7yg56Z6N180g0BwaBPj1hdtvuOMzjF5hS3SUd2/+3ZrOebKR0DlVUk8c7L8+8701n/Php8OTHVdlnJfJbHBURLaq6piIbAVCk/aVp6NJOxtOMeAF9TmZiyIiZGUtBbPR0mN/rpgktoElzvV2fEZz2RId5T+FGJDrDlUfTe3HJzrzHT93KGSAsPRZzeg6VnVrrJDZ4DbgmtL7a4Cau5kAyAVK1lf8NemHrNs1crTU4bFax6eqF6rqhRFW9W0bhtFDVDNyDs1sAFwH3CIi7wEOAlfWWnmOAkdzgqLMSU2r8NY6Cx3fddTZ8QF8c2xtuYqM3kGKy6/2AIdV9a2d1qcXWdU4r5LZ4LJGKvcJmCWHonjB+jQkIvJVYDewUUQOAR+lCR2fYXSY91Nc2TXQaUV6lY7uEJyXeU7KOHnmmMse76QqHUNVr17mUEMdn2F0ChHZAVwB/AXwxx1Wp2fpbGwNZ5ajc7/ED6axnXiGsWb4BPAhoL/TivQyHTXOio+qx9o0zAI4FjK0g4jIDcBbgXFVPa8kqyvswPZXpPkf9722Qn6dU/1qjZ/N37R6oRbwyaP72laXiCy09/0isnuFcosrjQZcs+FhdNQ4r10EkTiOxIs7BDvU9/xi/oudqbh7uBH4O4qxYRZY2H15nYhcW/r84Q7otla5FHibiLwFSAADIvIlVX1XeaHyJbZbY5vX4uisYWxY1yIcieM4MRzWYsyQ3sB2X7YfVf2Iqu5Q1V3AVcD3lxpmozps5NwCRKKkYqPE3QFSMrw2vTa9S9W7L8sfvU87bUMbVDOMF7CRcwsQovS7W9jEafQHA5ZDsEtR1RWDqJRv8tm0yVaE1Yqq3m1rnOvHjHOLcMTFUZsQ7EKq2n1pGJ1GioOHNlUmcgzIAL2+qHkj9d3D6aq6qdnKwGLbHix9rFe/bqLWewht21IkxX8pW63xt8BE2YTgiKp+aLWLl7XvWmjbalm415Z9b6HiuxtWf6doV/3h3912GmcAEdmjqhe2tdIm0+330O36VUMz7qF89yVwlOLuy28DtwCnUdp9qapVhwFaC21bLZ2+1/Vev00IGmsW231p9DLmEDUMw+hCOmGcr+9Anc2m2++h2/Wrhm69h27VqxV0+l7Xdf1t9zkbhmEYq2NuDcMwjC7EjLNhGEYX0lbjLCKXi8jjIvJkaY1p1yMiO0XkByLyqIg8IiLvL8lHROROEXmi9DrcBbr2XPtCMXqciIyLyMNlMmvfNtHp9l+tXUUkLiI3l47/orR2vVl1h/6+l5TZLSJTIvJg6e/PmlX/iqhqW/4AF3gKOBOIAQ8B57ar/gb03gpcUHrfD+wHzgX+Bri2JL8W+OsO69mT7VvS/Y3ABcDDZTJr33XQ/tW0K/CHwD+U3l8F3NzE+kN/30vK7Ka4kamt/5d2jpwvAp5U1adVNQ98jWKEsK5GVcdUdW/p/QzF1Dvb6b7oZj3ZvtAz0eN6tn1Xo8PtX027luvydeCyUuLphlnh991xGjLONT7mbQeeK/t8iC5phGopPU6dD/yCGqKbtYmeb98lWPt2lna1fzXtulhGi9k5poCmhwlc8vteyiUi8pCI/JuIvKzZdYdRt3EuZdf9NPBmio/5V4vIuc1SrNsQkT7gG8AHVHW6/JgWn32aviZxrfo4a6UV7WttWz2t+n53Eyv9voG9FONfvBL4FMUQAK3XqeRTqf1EkUuAP1fV3yp9/giAqv7VcuUdoj+NOMl6dUUQohrFFSHuCKmIT9QJSPfP4iS8xXL+bJwnT/SR0dpjliSdEbbFIeL4ZP0o+UDwAiEbKIEGeOIT4CPLJiRfnnwwfVyrDCBT6vz2A79BcTRxH3C1qj4aVt6VmFbTtgNOokJ22o5jlfVv2FUh0+MHKmR7D67+/Tmnb3DVMgBzhWiF7EghX9W5rWxbAEei6jqxCrkXzFWlXzuIO0Oh8lwwWcNVwiI6+IDuV9Wz69FrJUTkkhiJn6ac2lJVTQaV39nVcKTyu78SMandVgX4NZ+z3He3kdgaYY8jFUnWygOWi7hsTV1y6vGyTCFaurFymaPFwb2DQ5QYO3QTQ9EIZ/TBq4an2Zye5cJL7yX10qNoHrTgcPy+c/idb15QV862MxO/zp+9OMHGZIYnp4YZm49zLOvyxEyBrPpMyCQZZ7biPK3in3Jw9jthkbeWY9EXByAiC764UAMScZIrtu0ClyVeXCH7h49VboSSd/33Cpn+r3dXyKLvWf2+b7zg9ZXXCuncHjq6tUL2V4efXfX60Nq2BXCdGEPJ8yrkxzP311Btazk9uTtUvj/zz1VfI+KOVMg8/wTg3VqnWqtxX8rpZ3f6d2s66dszf19zRen4rprK73Ar/9+rMSczNZ+z3He35YGPtCxXWNwdrGqYrvi4GmVTsIkBSdAfcdkYdxiMKRdumGLH4Em2bT7K5pc/AcDsc5uZ+t7LefLg6dw3vplDc1Eelx/Xpe+zwS+56alfYyg2yI4UbIp7JNMBO1NCQaOMZ7cwU4DpQsDxQp4CHiecSbKSqau+FVi18yvv+NwaRwXrnKoGFsYpXNeKi6qqN+yOtuLSPU8jxvkwsLPs846SrCk4OGxwkmxJRtiSUM4enGFjYo7Xve5e+s5/nuDss/Be/ecE4z8n96HHeWD/2Xzr2Y3848kv4QezUMfjBcBM9glu50ki2SF+R6/m9LTPcCzH9oEpAA5PDzKZjzM2H+OJmTgZL0omyLbCOK9KPR2fUT3lnZ8jlS6N9YTWFlb1cuCTFJfJfV5VW2LY1zqNGOf7gLNE5AyKRvkq4J31XqzcpRHRCElNsT0d4cX9BTbE82zrmyEVy5GZGEJ+6RN5cpzoz64ld3SY+x57E788OcTBuRyBzlOvYT5Fm2CeQ7ksj04lGYzGGc8W/U8TuQhzvjDvC65AzHGI+zHimiQgwBOvKhdHFbS081vnVNW25Z1f1E1b51cFZQsFFv35InLbSv58I5y6jbOqeiLyR8B3KPaQN6jqIzVfZ4khi2iElPYxpH28fnSGS896jEisQDyZxcvFePiJszi499U8PpXkJ5OzzEqGY/I0WX+KucJxikslG0c1x8/zt7J3sh9XorgSxxGXiMRxibIreDHnpvtIuMK0l4QAspJlRooTvQv+3gYMdVM7vwWu++3vVMgy91eukhr6369ptKpFLr7njqrK/Z9b3lche3Oqcg7qjrknG1WpJW1rAHX4841wGvI5q+rtwO1N0gUoujPiGiPlRBiKzdI/MrV4zA8cjmT6eGomyQOTee7N3oJqtpnVl6F4/gSeP1EmE1xnANdJMhDfiK99OAJREeIapYC37NVqrr1JnZ9RST1tmyLNRVQmxbidxicE0/EXhcoLfrirLO8dCZUvN/G3o293hezQ7N2hZc9KvLFC9sz8XaFll6HmhQJJ6avl+uuGrsmEEhAAMBgM8fL0AMMxxVdh7LltPHN8Mz88soGTOXhmPssxZ4yj8jSqhTZrqQRBhkBzPFt4gKxkSGk/pzPK1liCo3lhsrTRqhmujVZ0fkYRa9vOUu4yGnZHzWUUQseN8wvLvQKUgDQxXtTnMRzziiPlyRF+Oj7CZ459Y9kRQztRPFCP+fyzHMw/S8TdwM74f2RDXMh4ERx18KUpPmfD6EVsrqRJdNw4Lx1hppwIW5M5BmI5Ds+lmZgcZP90cYKuGwk0z2Em0NkNnNQ5fLdAQIBj0ViN9Yn585tEx41zOYLDUMzlFVsOE4vlufW+8/j67D9T8DP4wdIdld1BEMyyb/4OHndiJKMbGWQ7grPopumkkQ5zrRRylUvCNn2u4Qm2pvCpI5+ukJ2drowtpPZk0rXYXEnz6CrjDOCKEIl4RFyf6ULAfL66HWKdQwk0Q+BnyDvJLmxRw2gv5s9vDl1jShZGmDOFgGeObybq+Jz0mrMsrl2IOESIl/zO7Z6sNFpJ0hVeOli5Hf72GvYeLbcqoy8SHhLkWD58jiUZOy1UvjUaHiztSGFfFdoViWvlTtN64sgYjdM1xnmBfBAwlknjOgGzdKefeTkEh4hGcEpujSZtRjGMNc1ossD7z6ttzvCuB86quZ5+d0tN5f06lsb+b0Mvqfmcv5yt3HsAXWicp4MCT80mcAQmnc6vzqgFEQcHB8GxVRuGYTRE1xnnY84EPz05RIByTLtjoqoWBKdrVmqM+pVR3t7y9bBH6O7dvBXRkK+oPWUb64CuMc6Ci6MOSsCszOGJh1fIdVqtmtHFdRqGYRj10xXGWXCJa4K4JvDxmJc5CpLDD3prQtAP8szJTFeNno3m0BfxeOPmiQr5//d89dfI5J4KlbsS/jNUDR+cxNzw7c7TjIfK3ztyRYXsE2OVyxYBHpz7aojU3HOdoGssSExjxDWGSwRPPAqa67kJtUAL5HUOT3M2fjYMoyG6YuQMkNIkw06SbBArRXiLccKJ00sL0lQ9csFsMXKdROmivs8wjB6jK4yzow6DkmBLIsK85yJ5YU7jjDlJWhVzrhUEQZ6cP40ncWKSWvZxtV28YagyL9veyd7y4z/lVSZC3hw7pwOaGEZ76ZqhXUSEqAMikMdnXnL4bY861xiKjx/kCHpMb8Mwuo9VjbOI3CAi4yLycJlsRETuFJEnSq/DjSnhEHUcki5kfWWf7OXxwo+YX2aHVLeimsPzTpL3pvB7yiFjGEa3Uc1z943A3wFfLJNdC9ylqteJyLWlzx9uRBEBIqL4qkznDi0Jct8rKIpHoLm6UqQb3Utfao6Lz3+w8sADjV97Ovt4TeW9INw1NTUfvl793a+vXN3xibGaqjQ6wKojZ1W9B1ia3PHtwE2l9zcB72hUkbgjJCNKzHEQ6RpvS0cRkQMi8isReVBE9nRaH8Mw2ke9M1abVXWh7z0CVCahqwEHh5grpCM+Cbcr5ii7iTep6vF6ThxNVI7e757/QsMKtZNs/lCFrC9SkfXIMNYcDVtCVVURWTbNTHmuMFcqI16tVRzcZiR5NYw1T9+2OS79WIjLaAXmL5+puZ6Z7BM1ld+YfnXNdXzoP9bu5/rLT4bL6/UfHBWRrQCl1/CtSRRzhanqhap6oSuVgd6hmD/QVyUfCF7Q++nEBBeXKHFNhMeGqB4Fvisi95c6uVPrEXmviOwRkT1+k7KOG4bRHdRrnG8Drim9vwa4tVFFAoVAZc3sq3Mluhg+tAFer6oXAG8G3icip6RGrqbjM8Ixf35rEJGdIvIDEXlURB4Rkfd3WqdeZdVhnYh8FdgNbBSRQ8BHgeuAW0TkPcBB4MpmKOOI4qyFkGPikNJ+BrSPWebIS74u14aqHi69jovIt4CLgHuarO16pmp/vtuXY+TSkCiJN4SXj7iVq0s9/2RNyi1Hwh0IlS8X93/DxqXz+S3FAz6oqntFpB+4X0TuVNXuDX3YpaxqnFX16mUOXdZMRRxZO5EgHYnQF/Qz6CTwgoAZas9/KCJpwFHVmdL73wQ+Vss1csFaadFTOV02VsgsSV13UFooMFZ6PyMi+4DtdHNc2i6la5ZGFALIB0JBFdXedm6oBuQkRzZIUpDasymU2Ax8S0Sg+H/6iqre0SwdjUV/vgKfVdXrO63QWkNEdgHnA5V78I1V6RrjPOcHTORcZrwC9LjnWdXjhByhIPm601Wp6tPAK5uvnVHi9ap6WERGgTtF5LHSmv5FylcanbahEyr2LiLSB3wD+ICqVjw6ntK2o5W5GY0uia0REFAIArI+5NTv/ZEzPjmdZV4y5KS38iCuF8r9+cCCP39pmcUJ1419a9NF1ApEJErRMH9ZVb8ZVuaUtg1JnGt0eOQsuIsrGk7qPIW5gKPOcVTrdgV0BaoFZvLPM+dMkIpsoN8Jz67cau4aX5vxPR6R8KD11dIMf74RjhT9cF8A9qnqxzutTy/TUePsqINL0ThPOZNM4DGlR9CeDxrkU/COUaCYUzAd22CZUbqL2v35rkA6HnJgLrT462L/oUJ2T5N2Z3rLZEhZjr/43htDpL9qii4hXAr8HvArEVnYWfInqnp7qypcq3Tc57wwcnYoxtSISNgPwDCah/nzW4eq/pi1s/Cqo3TUOEeIkNQkbmlU6aiQc7N0iSvcMAyjY3R85CwITiklKhRdHYZhrCNcl2BgqKZT3hS/vOZq7pyrbbXkVj2z5jr++z/VE5TrU6HSjhvnBU44k8zJDFP+EejxCcFuYWssxEVki0cMoyfoCuMcoMzKFJPec+S8GZTeD35kGIbRCB1eSueQJIYDKAE5bwY/mKfXN6EYa5CCEhypfhXRtISv4mgGWW8yVH52+u2h8s8c/XSFbFffb4WWPTD7nfoVM5pKR41zTGP0u1EiIvhBgbx3jKJhtpGzYRjrm467NSIiuIsLbywovWEYBnTYOAcEzPs+jggFsp1UZU0y5VlnZxi9SmeNsxRjajgIan5mwzCMRTo+cs5SAAWP2rakGoZhrGWqyYSyE/gixXgEClyvqp8UkRHgZmAXcAC4UlVrSvVQIM+0zBZH0F7rZrcNo2EEnGT1T3eP5L/XMlX8IHwgs9zTpyPpCtkbYi8OLXsAW63RLVSzHW8h7cy5wMUUc9mdC1wL3KWqZwF3lT7XTCABwTp2aYjIDSIyLiIPl8lGROROEXmi9FqZ88gwjDXNqsZZVcdUdW/p/QywkHbm7cBNpWI3Ae+ovXKHqMaIawJh3cZ0vRFYuhe1KR3fRDBX8WcYRm9QUyCLJWlnNpfyhQEcoej2CDvnvSKyR0T2+JqvOB5Rl4i6OLI+Y2qUsm8szcDZcMdnGEZvU/WE4NK0M6VYuACoqpZysVVQys12PUDcHawo44mPEhAE69e1EUJVHZ9hrAUkvYXIaz5Y0zk/zH+45no2py+uqXxQR0am6ULzoqVWZZyXSTtzVES2quqYiGwFxmutPCAgJ8X1zYH2eoD91rBSx1eeh82VRFv1Wm/o0DDzV1wRcuQroeUL3rEKmSzzP9qQelmo/Hjm/qr1A5jQQ6HykdQ5FbJ7c+Flje5hVV/CCmlnbgOuKb2/Bri11sp9KZCTLHMyQ2CR6Mo5WurwWKnjK8/D5kqsrQoahtFaqhk5h6adAa4DbhGR9wAHgStrrTyrs0zmn8UP5vGDTK2nr2UWOr7rqLPjA7hv/h+bqZNhVI2IuMAe4LCqvrXT+vQiqxrnVdLOXNZI5Z7myHsnUF2/W7dF5KvAbmCjiBwCPkoTOj7D6DDvp7iya6DTivQqHQ98tN5R1auXOdRQx2cYnUJEdgBXAH8B/HGH1elZ1uf6NcMwWskngA+xQmD28iW2x45Nt0+zHqKjI2dXorhOEr/0LxRclAIash7aMGpFRG4A3gqMq+p5JVldYQec6CCpbWEB6sNXa4SxnPvueGZv1dcA8PyJUPnUMuFpzo+9pUJ2f+7bNdVZLSKy0N73i8ju5cqVL7G98MIzLYB7CJ0Nti8pRhJnEeDjEsUlyqw/zkz2KRRbvdEoYZkxHs/UNbfYq9wI/B3F2DALLOy+vE5Eri19rn3RrLEclwJvE5G3AAlgQES+pKrv6rBePUdH3RouEZIyQJ9spJ8N9DNCwh2Edbpb0Ggutvuy/ajqR1R1h6ruAq4Cvm+GuT66YkJQ8ZnmJLlglpw/jaoFiTdaRtW7L8s3+Zx22oY2qGYYL9AVQ9SAgJnCEabn95PNj2Hpqox2oKrKCgkryzf5bNpkK8JqRVXvtjXO9dMVxhmKsWgVxTJvGy2mqt2XhtFppDh4aFNlIseADHC8bZW2ho3Udw+nq+qmZisDi217sPSxXv26iVrvIbRtS5EU/6VstcbfAhNlE4Ijqvqh1S5e1r5roW2rZeFeW/a9hYrvblj9naJd9Yd/d9tpnAFEZI+qXtjWSptMt99Dt+tXDc24h/Ldl8BRirsvvw3cApxGafelqi6dNGypXr1Cp+91vdffFROChtEKbPel0ct0jc/ZMAzDeIFOGOfrO1Bns+n2e+h2/aqhW++hW/VqBZ2+13Vdf9t9zoZhGMbqmFvDMAyjC2mrcRaRy0XkcRF5srSMqesRkZ0i8gMReVREHhGR95fkIyJyp4g8UXod7gJde659oRigSETGReThMpm1b5vodPuv1q4iEheRm0vHf1FaHtmsukN/30vK7BaRKRF5sPT3Z82qf0VUtS1/gAs8BZwJxICHgHPbVX8Dem8FLii97wf2A+cCfwNcW5JfC/x1h/XsyfYt6f5G4ALg4TKZte86aP9q2hX4Q+AfSu+vAm5uYv2hv+8lZXZTXCvf1v9LO0fOFwFPqurTWowJ+jWKQWi6GlUdU9W9pfczFLM7bKf7Auj0ZPtCzwQo6tn2XY0Ot3817Vquy9eBy0q5TRtmhd93x2mncd4OPFf2+RBd0gjVUnqcOh/4BTUE0GkTPd++S7D27Sztav9q2nWxjKp6wBTQ9EhUS37fS7lERB4SkX8TkfB06U3GNqFUiYj0Ad8APqCq0+Udt6qqiNiylxZh7dtZ1kP7L/19Lzm8l+IW69lSnOpvA2e1Wqd2jpwPAzvLPu8oyboeEYlS/Md9WVW/WRJ3WwCdnm3fZbD27Sztav9q2nWxjIhEgEEgPB1MHSzz+15EVadVdbb0/nYgKiIbm1X/crTTON8HnCUiZ4hIjKJj/7Y21l8XJd/WF4B9qvrxskO3AdeU3l8DdDrFSE+27wpY+3aWdrV/Ne1arsvvUAzg35SR/Aq/7/IyWxZ83CJyEUW72bTOYVnaOfsIvIXibOhTwH9r9+xnnTq/nmLM318CD5b+3kLR53UX8ATwPYrRzTqta8+1b0nvrwJjQIGiz/E91r7rp/3D2hX4GPC20vsE8E/Ak8C9wJlNrHu53/cfAH9QKvNHwCMUV5L8HHhdO/4vtkPQMAyjC7EdgoZhGF2IGWfDMIwuxIyzYRhGF2LG2TAMowsx42wYhtGFmHE2DMPoQsw4G4ZhdCFmnA3DMLqQ/x8NHNoaYXzZ1AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 12 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "f, axarr = plt.subplots(3,4)\n",
    "FIRST_IMAGE=0\n",
    "SECOND_IMAGE=7\n",
    "THIRD_IMAGE=26\n",
    "CONVOLUTION_NUMBER = 1\n",
    "from tensorflow.keras import models\n",
    "layer_outputs = [layer.output for layer in model.layers]\n",
    "activation_model = tf.keras.models.Model(inputs = model.input, outputs = layer_outputs)\n",
    "for x in range(0,4):\n",
    "  f1 = activation_model.predict(test_images[FIRST_IMAGE].reshape(1, 28, 28, 1))[x]\n",
    "  axarr[0,x].imshow(f1[0, : , :, CONVOLUTION_NUMBER], cmap='inferno')\n",
    "  axarr[0,x].grid(False)\n",
    "  f2 = activation_model.predict(test_images[SECOND_IMAGE].reshape(1, 28, 28, 1))[x]\n",
    "  axarr[1,x].imshow(f2[0, : , :, CONVOLUTION_NUMBER], cmap='inferno')\n",
    "  axarr[1,x].grid(False)\n",
    "  f3 = activation_model.predict(test_images[THIRD_IMAGE].reshape(1, 28, 28, 1))[x]\n",
    "  axarr[2,x].imshow(f3[0, : , :, CONVOLUTION_NUMBER], cmap='inferno')\n",
    "  axarr[2,x].grid(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8KVPZqgHo5Ux"
   },
   "source": [
    "EXERCISES\n",
    "\n",
    "1. Try editing the convolutions. Change the 32s to either 16 or 64. What impact will this have on accuracy and/or training time.\n",
    "\n",
    "2. Remove the final Convolution. What impact will this have on accuracy or training time?\n",
    "\n",
    "3. How about adding more Convolutions? What impact do you think this will have? Experiment with it.\n",
    "\n",
    "4. Remove all Convolutions but the first. What impact do you think this will have? Experiment with it. \n",
    "\n",
    "5. In the previous lesson you implemented a callback to check on the loss function and to cancel training once it hit a certain amount. See if you can implement that here!\n",
    "\n",
    "练习\n",
    "\n",
    "1. 尝试修改卷积层参数。将32改为16或64。这对准确度和/或训练时间有什么影响？\n",
    "\n",
    "2. 删除最后的卷积层。这将对精度或训练时间产生什么影响？\n",
    "\n",
    "3. 增加更多的卷积层会有什么影响？实验一下吧。\n",
    "\n",
    "4. 除第一项外，删除所有的Convolutions。这样做会有什么影响？请完成实验。\n",
    "\n",
    "5. 在上一节课中，实现了通过一个回调函数来检查模型的损失，并在损失减小到一定量时取消训练。看看是否能在这里实现？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "height": 415
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 50716,
     "status": "ok",
     "timestamp": 1550247480521,
     "user": {
      "displayName": "Laurence Moroney",
      "photoUrl": "https://lh3.googleusercontent.com/-RcxktLY-TBk/AAAAAAAAAAI/AAAAAAAAABY/b4V4dTIqmPI/s64/photo.jpg",
      "userId": "06401446828348966425"
     },
     "user_tz": 480
    },
    "id": "ZpYRidBXpBPM",
    "outputId": "70c1f9f2-880c-4923-9887-8f1d4c6b8383"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2.0\n",
      "Epoch 1/10\n",
      "1875/1875 [==============================] - 17s 9ms/step - loss: 0.1538 - accuracy: 0.9537\n",
      "Epoch 2/10\n",
      "1875/1875 [==============================] - 17s 9ms/step - loss: 0.0519 - accuracy: 0.9843\n",
      "Epoch 3/10\n",
      "1875/1875 [==============================] - 16s 9ms/step - loss: 0.0345 - accuracy: 0.9890\n",
      "Epoch 4/10\n",
      "1875/1875 [==============================] - 17s 9ms/step - loss: 0.0226 - accuracy: 0.9934\n",
      "Epoch 5/10\n",
      "1875/1875 [==============================] - 17s 9ms/step - loss: 0.0154 - accuracy: 0.9955\n",
      "Epoch 6/10\n",
      "1875/1875 [==============================] - 17s 9ms/step - loss: 0.0105 - accuracy: 0.9965\n",
      "Epoch 7/10\n",
      "1875/1875 [==============================] - 17s 9ms/step - loss: 0.0086 - accuracy: 0.9972\n",
      "Epoch 8/10\n",
      "1875/1875 [==============================] - 16s 9ms/step - loss: 0.0065 - accuracy: 0.9978\n",
      "Epoch 9/10\n",
      "1875/1875 [==============================] - 17s 9ms/step - loss: 0.0057 - accuracy: 0.9981\n",
      "Epoch 10/10\n",
      "1875/1875 [==============================] - 17s 9ms/step - loss: 0.0039 - accuracy: 0.9987\n",
      "313/313 [==============================] - 1s 3ms/step - loss: 0.0580 - accuracy: 0.9866\n",
      "0.9865999817848206\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)\n",
    "mnist = tf.keras.datasets.mnist\n",
    "(training_images, training_labels), (test_images, test_labels) = mnist.load_data()\n",
    "training_images=training_images.reshape(60000, 28, 28, 1)\n",
    "training_images=training_images / 255.0\n",
    "test_images = test_images.reshape(10000, 28, 28, 1)\n",
    "test_images=test_images/255.0\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)),\n",
    "  tf.keras.layers.MaxPooling2D(2, 2),\n",
    "  tf.keras.layers.Flatten(),\n",
    "  tf.keras.layers.Dense(128, activation='relu'),\n",
    "  tf.keras.layers.Dense(10, activation='softmax')\n",
    "])\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(training_images, training_labels, epochs=10)\n",
    "test_loss, test_acc = model.evaluate(test_images, test_labels)\n",
    "print(test_acc)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Course 1 - Part 6 - Lesson 2 - Notebook.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
